ABS COMPILATION METHODS

4.11 In 2001, when the International Merchandise Trade, Australia: Concepts, Sources and Methods (cat. no. 5489.0) was first published, ABS statistics were processed on a mainframe computer. Between 2003 and 2006 the international merchandise trade system was re-designed, tested and transferred to a new platform. Over a similar period the DIBP were re-engineering their systems for collecting exports and imports information. While these system changes were significant in terms of the way data are collected, delivered and processed there was only minimal impact on statistical output (Appendix 2 in the 'Downloads' tab). If you are interested in the previous systems and methods please read the 2001 version of International Merchandise Trade, Australia: Concepts, Sources and Methods (cat. no. 5489.0).

OVERVIEW OF THE INTERNATIONAL MERCHANDISE TRADE PROCESSING SYSTEM

4.12 A high level depiction of the compilation of international merchandise trade statistics is shown in Chart 4.3. The chart shows the movement and processing of exports and imports information from the DIBP system to the release of ABS statistical outputs. In most cases the data moves in one direction but the two-way movement of data between the ICS and the ABS environment reflects the normal transfer of the DIBP information to the ABS and the occasional transfer of information from the ABS to the DIBP, e.g. delivery of new export and import statistical codes. The movement of data in both directions between the data load, IDW, edit/amendment system and aggregate and output editing reflects the interaction, amendment and re-checking of the data between these parts of the International Merchandise Trade System (IMT system).

4.13 The IMT system uses standard ABS statistical infrastructure and tools with adaptations to cover the type of data (export and import declarations, not survey questions and answers) and the volume of data received on a daily basis.

4.14 The IMT system has a separate test environment for testing changes or new functionality prior to implementation in the production environment. The test environment is also used as a training environment.

4.15 In reality the system used to receive, process and disseminate the statistics is much more complex than shown in Chart 4.3, which is only a representation of the key processes. The following sections describe the key processes and stages culminating in the release of monthly international merchandise trade statistics on the ABS website.


START THE PROCESSING MONTH

4.16 The timing of the processing cycle differs between exports and imports due to exports being compiled on a departure date basis and additional time allowed for the finalisation of export manifests. Consequently, for exports the commencement of the new processing month normally occurs around the middle of the reference month. For imports the commencement of the new processing month happens earlier, generally on the sixth working day.

4.17 Threshold edit values are set to manage processing workloads. These are used to ensure that all records with values above the thresholds and records with fatal or warning error messages are checked by ABS editors.

4.18 Before processing commences it is necessary to ensure that metadata changes taking effect from the current reference month have been implemented correctly. For example, if a new 10 digit commodity code is active in the ICS from 1 January 2014 this new code must be loaded to the IDW metadata before processing of January 2014 data commences.


INPUT DATA WAREHOUSE (IDW)

4.19 The IDW is a key database in the processing of international merchandise trade data. It is the repository for all export/import data and metadata, including:

• all the source data (including all versions of each transaction) from the DIBP
• all amendments
• information added by ABS editors about the amendments
• all the metadata used by the IMT system (e.g. commodity, country, port codes and descriptions) and some metadata which can be used to link international merchandise trade data to other ABS economic surveys, e.g. ABN
• additional transactions which are added when customs data are not available, see paragraphs 4.55-4.56 in Other Data Sources.

EXPORTS AND IMPORTS DATA LOAD (TO THE IDW)

4.20 The data load processes initially reformat and map some customs data items to ABS data items. The most important re-mapping is for any fields which use the UN/Locode (i.e. the country, port and state fields). This is because when the DIBP introduced the UN/Locode in their system (with the introduction of the ICS) the ABS decided to minimise changes to the IMT system and statistical output by continuing to use the previous country, port and state codes (see Diagram 6.1 in Country and Overseas Port Classifications and Processing).

4.21 Processing stage codes are added to records in the data load to IDW. These codes are used throughout the IMT system to identify particular record types and their status at particular processing points.


OUT OF SCOPE RECORDS

4.22 The load process identifies out of scope transactions which are not edited but are still loaded to the IDW for analytical purposes. The following record types are out of scope of international merchandise trade statistics.

• Consular records: These are transactions covering goods sent to Australian diplomats overseas (these do not subtract from Australia's stock of material resources) or brought to Australia for foreign diplomats stationed here (these do not add to Australia's stock of material resources). These transactions are internal flows and are out of scope for international merchandise trade statistics (see paragraph 2.47 in Goods Excluded from Australia's International Merchandise Trade Statistics).
• Customised software records: Some goods transactions are considered services. Where possible these are identified in the load, e.g. computer software with a calculated unit value (reported customs value divided by reported quantity) greater than $10,000 is an indication that the software is customised and not an off the shelf product (see paragraph 2.49 in Goods Excluded from Australia's International Merchandise Trade Statistics).
• Value below the threshold: The customs value thresholds which require the lodgement of a customs declaration are described in paragraphs 2.6-2.8 in Value Thresholds for Customs Declarations. Import consignment lines which are below the value threshold ($1,000) are often received because they are part of a consignment of goods which includes lines above the threshold or because the goods for import are tobacco and alcohol which require a full declaration regardless of value. These low value lines are treated as out of scope. Export consignments with values below the consignment threshold ($2,000) are also received as an export permit may apply to the commodity. These are also treated as out of scope. This treatment ensures a consistent approach as the records identified as out of scope only represent a small proportion of the value and volume of goods below these thresholds. Value estimates for goods below the thresholds are included in Australia's balance of payments statistics.
• Outside the revision period: Amendments for records which are outside the ABS revision period (see paragraphs 10.15-10.16 in Dissemination Policies and Practices).
• Bank notes in circulation: These are out of scope when they are valued at their face value and represent a financial transaction (see paragraph 2.46 in Goods Excluded from Australia's International Merchandise Trade Statistics). As these records are not easily identified in the load process it is more likely that they are changed to out of scope by an ABS editor.

4.23 Replacement or amendment records which alter an out of scope record to an in-scope record are edited and included in international merchandise trade statistics. Chart 4.4 illustrates the scope of international merchandise trade statistics, that is, international merchandise trade statistics equals customs declarations and some bunkers, and excludes international non-merchandise trade and out of scope records.




EXPORTS AND IMPORTS EDIT

4.24 Once the export and import data are loaded to the IDW, all in-scope records are passed through their respective edit process (exports or imports edit). The edit process checks the details of each transaction against a set of edit conditions which are categorised as fatal, warning (notifiable) or informative. Transactions which fail fatal or warning edits then have messages sent to the edit/amendment system for investigation by an ABS editor. The investigation may require contact with the exporter, importer or agent to resolve. Informative edits usually indicate an automated system adjustment has been applied. Transactions which fail informative edits are not viewed by an ABS editor unless they also contain a fatal or warning edit.

4.25 The following list describes the most common edit checks applied to both export and import data.

• Identification of all transactions with values equal to or above a large value threshold. These transactions are subject to detailed checks including confirmation with the exporter, importer or their agent if necessary.
• Legality checks to ensure that valid and up-to-date codes are reported and that all mandatory fields are completed.
• Internal consistency checks, e.g. if the mode of transport is reported as sea, the Australian and overseas ports should not be airports.
• Unit value and for exports gross weight unit value edits are applied. These edits are generally informative and are only assigned to editors if they fail a warning or fatal edit.
• Automatic adjustment of certain import records, where the unit values fall well outside the expected range for the relevant HS classification. The quantity is assumed to be in error and is estimated.
• Tailored edits for specific commodities and/or countries and/or states. Examples include: checks of minerals which are only mined in certain states, prohibited or illegal imports and state of exports for commodities not manufactured in Australia.

4.26 To ensure that ABS editors focus on the most significant transactions, and to minimise contact with exporters (or their agents), significance editing is used in conjunction with many of these edit checks. Significance editing gives a score to all transactions to indicate how the reported value (exports FOB value, imports customs value) differs from a predicted value. The size of the score is an indication of the potential error. Transactions with scores beyond the significance score thresholds are flagged for review by an ABS editor. The score is allocated based on data calculated by the input editing benchmarks. The input editing benchmarks use the previous 12 months of data to determine if quantity or gross weight is the best predictor of value.

4.27 The re-edit process is essentially the same as the edit process but occurs following editor amendment whereas the edit process occurs when a new transaction (or revised version of a transaction) is received from the DIBP.

4.28 Chart 4.5 shows the imports data load where the IDN information from the DIBP is re-formatted and loaded to the IDW. Earlier versions of an IDN which are already on the IDW are identified (to ensure that the new version is not loaded as a new transaction) and up-dated in the Load to the IDW process. In Chart 4.5 this is shown as data moving from the IDW back to the Load to IDW stage and then through to the IDW again. The metadata which is used in the Reformat and Imports edit processes includes the edit checks and country/port metadata. After the edit check a new version of the transaction with any relevant alerts is created in the IDW (shown as a two way arrow between the IDW and Imports edit stages).




4.29 There are many similarities in the exports and imports data load processes. However, they are separate processes involving different data items and metadata and there are some aspects which are unique to the exports load. The differences in the processes can be seen by comparing Charts 4.3 and 4.4. The unique export processes are described in the following paragraphs.


EXPORTS MATCHING PROCESS

4.30 Exporters must complete an export declaration with the DIBP and obtain an Export Declaration Number (EDN) in advance of the goods arriving at the wharf, or airport, for export. The EDN is an intention to export and includes preliminary information about when the goods will leave Australia and on what carrier the goods will be transported. The actual departure details are listed on the ship or aircraft manifest. When the ABS receives both the EDN and its manifest, the transaction is said to be acquitted 'matched' and the goods exported.

4.31 The exports matching process matches EDNs and manifests and adds the departure details from the manifest to the EDN information. In most cases the EDN is received by the ABS before the manifest. The EDN will be loaded to the IDW and identified as an EDN awaiting acquittal. Once the corresponding manifest is received the EDN is acquitted and, if the transaction has not been reported in Australian dollars, the RBA exchange rate for the day of departure will be used to convert values to Australian dollars. If the manifest is received first it is also loaded to the IDW but identified as a manifest record. When the EDN is received, the matching process retrieves the manifest from the IDW and the EDN is acquitted. This process of retrieving EDNs/manifests from the IDW and matching them with their corresponding EDNs/manifests is depicted with a two way arrow between the exports matching process and the IDW.

4.32 A small number of records are 'self acquitting'. This means the EDN is marked as acquitted (and therefore exported) without being shown on a manifest. Self acquitting EDNs are those export declarations which do not get recorded on a manifest. These are postal records, ships and aircraft exported under their own power, ship and aircraft stores and spares and unaccompanied baggage, identified by the relevant export goods type.

4.33 Because EDNs are lodged and passed to the ABS prior to export, the exchange rate can be for a future date and is therefore unknown. When these EDNs are loaded to the IDW the FOB value is estimated using the latest available exchange rate. To update these estimated records the load process checks for new exchange rates and retrieves the estimated records from the IDW. These records are passed through the load process again to update the information. In Chart 4.6 this is shown as data moving from the IDW back to the Match and Load to IDW stages (with input of exchange rates) and then through to the IDW again.




FINALISING MONTHLY EXPORTS DATA

4.34 Towards the end of the processing month, a report is produced showing all unacquitted (no manifest received) large value EDNs with expected departure dates within the processing month. If the report shows an abnormally high value of unacquitted EDNs sometimes a decision to load more daily files to the IDW is made in the hope that manifests to acquit the EDNs are received. However, there comes a point where meeting the publication deadline means that no more data is loaded. At this point the monthly exports telephone follow-up occurs.

4.35 The follow-up involves direct contact with exporters (or their agents) about their large value unacquitted exports to determine if the goods have actually been exported. Goods that have been exported are acquitted using the manifest information supplied by the exporter and the transaction is included in monthly estimates. If the goods are not exported the expected departure date is changed to the following month and the goods are not included in the monthly estimates. Because ABS resources are limited, and experience shows that most EDNs with expected departure dates in the current processing month have been exported, all records below a given value (normally around $1 million) are included in the monthly estimates without checking their actual departure.


EDIT/AMENDMENT SYSTEM

4.36 The edit/amendment system is used to view and, if necessary, amend transactions that have been flagged by the edit process as requiring examination by an ABS editor. The system is also used to view and, if appropriate to do so, amend unusual transactions identified as requiring ABS editor attention (e.g. by output edit processes). Amendments are applied as a result of contact with the exporter or importer, identification of a repeated mis-reporting error or other information provided in the transaction or similar transactions. Any amendment of a transaction requires the ABS editor to include comments (which are appended to the transaction) to explain the change(s) made. All amended transactions go through the re-edit process to ensure data integrity and transactions failing re-edit checks are returned to the edit/amendment system for re-examination. The system also provides the means to view any transaction using its unique identifier. The system is also used by ABS editors and managers to monitor work flow, e.g. the number of transactions awaiting editor attention (because they have failed fatal or warning edits).


AGGREGATE AND OUTPUT EDITING

4.37 Aggregate editing is an important component of statistical compilation and finalisation of output for release. Aggregate editing involves creating monthly totals by particular levels of commodity (HS 6 digit or SITC 2 digit), country, state, mode of transport and unit of quantity and comparing these to previously released aggregates. Significance scoring (scoring the current month aggregates against a benchmark derived from the previous 12 months of data) is used to highlight the most significant aggregate movements and to set workloads. Where necessary further follow-up with exporters and importers (or their agents) is undertaken and transactions amended. Records of any investigations and findings are kept and reviewed by more senior staff during the finalisation of outputs.

4.38 Output editing involves examining the data for particular commodities or at the 2 digit level of the HS. Examination of data at these levels may reveal a significant change. When this occurs the transactions contributing to the change are identified and scrutinised.

4.39 The trade query tool is an integral component of aggregate and output editing. The query tool allows staff to retrieve data from the IDW using a defined set of criteria. Simple reports can be produced, data aggregated and sorted to help ABS editors to understand movements or to resolve mis-reporting errors in the data. The trade query tool is also used in editing and verifying individual transactions. Where mis-reporting errors are found the relevant transactions are amended using the edit/amendment system.

4.40 International merchandise trade statistics are based on a large number of individual transactions and are released at a very detailed aggregated level, e.g. 8 digit exports and 10 digit imports by Australian port, country and overseas port. When ensuring the quality of international merchandise trade statistics, most resources are focussed on the data required for macroeconomic statistics.


OUTPUT FOR BALANCE OF PAYMENTS, NATIONAL ACCOUNTS AND PRICES

4.41 Once editing is finalised and prior to the public release of international merchandise trade statistics, data are prepared for inclusion in the balance of payments.

4.42 The preparation of data for the balance of payments signals the first approval stage in the release of international merchandise trade statistics. The export and import data supplied for the balance of payments includes totals on a SITC and, for imports only, a Balance of Payments Broad Economic Categories (BoPBEC) basis, information about significant month to month movements and other information used to compile goods data on a balance of payments basis. For more information regarding classifications such as SITC and BoPBEC and how data is prepared for the balance of payments see the Classifications and Balance of Payments Statistics sections.

4.43 Import data are delivered to the Balance of Payments Section on the 6th working day after the reference period, and export statistics are delivered on the 13th working day after the reference period. The later delivery of export data is due to the compilation of export statistics on a departure date basis.

4.44 International merchandise trade statistics are first released with international trade in services on a balance of payments basis in International Trade in Goods and Services, Australia (cat. no. 5368.0). Goods credits and debits on a balance of payments basis are closely related to international merchandise exports and imports but have been adjusted to take account of the principles of economic ownership and residence (see paragraphs 12.4-12.7 in Balance of Payments Statistics).

4.45 Australia's balance of payments is published quarterly in Balance of Payments and International Investment Position, Australia (cat. no. 5302.0). In this publication, the quarterly estimates of goods and services credits and debits are derived from the months published in the corresponding release of International Trade in Goods and Services, Australia (cat. no. 5368.0) with any revisions kept to a minimum. The quarterly exports and imports statistics which are included in the Expenditure on Gross Domestic Product estimates in the publication Australian National Accounts: National Income, Expenditure and Product (cat. no. 5206.0) are the same as those published in the quarterly Balance of Payments and International Investment Position, Australia (cat. no. 5302.0), meaning they include both goods and services on a balance of payments basis. Detailed exports and imports data on an international merchandise trade basis are supplied to the National Accounts Branch for use in compiling the annual supply-use tables, for input-output analysis and for other analytical purposes.

4.46 On a quarterly basis, data are supplied to the Prices Section of the ABS. The data includes detailed HS commodity level data (8 digit exports and 10 digit imports) by country, value, quantity and gross weight. The data are used in the compilation of the International Trade Price Indexes, Australia (cat. no. 6457.0).

4.47 International merchandise trade statistics provided to areas of the ABS which compile the balance of payments, national accounts and prices statistics include aggregates and some transaction level data so that the compilers are able to fully understand and interpret the data for their statistics. This is reflected in Chart 4.3 where the data provided for these areas are shown as coming from the IDW (transaction level data) and from the ABSIW (aggregate level data).


OTHER SPECIAL REPORTS

4.48 Any special reports produced for users from the IDW are aggregated and confidentialised in a separate process to the confidentialisation process described in the following paragraphs. Separate aggregation and confidentialisation is necessary because the reports contain non-standard aggregations of commodities and/or time periods, or are released (with consent) after a period of time has elapsed (see paragraphs 9.47-9.52 in Special Data Releases). These reports are produced on an ad hoc basis in response to user needs and are prepared on as a 'user pays' service.


CONFIDENTIALISATION

4.49 In Chart 4.3 the process of confidentialising international merchandise trade statistics is depicted as a single box which represents a database and set of utilities. The utilities enable:

• access to transactions to check the validity of a request to confidentialise data, i.e. if the trade related to a single business is identifiable
• investigations to determine the most appropriate restriction to apply
• changes, additions or removal of confidentiality restrictions
• the generation of reports like the monthly Confidential Commodities List published in International Merchandise Trade: Confidential Commodities List (cat. no. 5372.0.55.001)
• the provision of data for updating the IDW metadata so that the restrictions can be applied when outputs are generated.

4.50 Once the monthly exports and imports data are finalised and provided to the Balance of Payments Section, the output files for release to the public are prepared. It is in the creation of these output files that the confidentiality restrictions are applied to the data. Separate output files for exports, imports and import clearances are created.

4.51 The Data Confidentiality section explains the ABS confidentiality legislation and how confidentiality is applied to international merchandise trade statistics.


ABS INFORMATION WAREHOUSE (ABSIW)

4.52 The ABSIW is the repository for data available for dissemination, including international merchandise trade statistics. Also stored on the ABSIW are all the metadata which make the statistics meaningful, e.g. international merchandise trade commodity, country codes, descriptions and associated information. The ABSIW was created to enable all ABS data to be captured, defined and delivered in a consistent way.

4.53 At the end of the processing month after the data have been finalised and the output files created, international merchandise trade statistics are loaded to the ABSIW User database (an intermediate database where data are checked prior to release). The publication tables, time series spreadsheets and data cubes found on the ABS website are all extracted from the ABSIW User database. At 11.30am (Canberra time) on the day of release all storage tables involving international merchandise trade statistics and metadata for the particular collection, time series spreadsheets and data cubes are copied from the ABSIW User database to the ABSIW Output database (output environment) and ABS website.

4.54 Consultancy and subscription data are created from the ABSIW Output database from 11.30am (Canberra time) on the day of release of the publication.